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Interactivity is an important indicator of an educational app's reception. Since 
most educational apps are multimodal, it justifies a methodological initiative to 
understand meaningful involvement of multimodality in enacting and even 
amplifying interactivity in an educational app. Yet research so far has largely 
concentrated on algorithm construct and user feedback rather than on multimodal 
interactions, especially from a social semiotics perspective. Drawing from social 
semiotics approaches, this article proposes a multimodal analytic framework to 
examine three layers of mode in engendering interaction; namely, multiplicity, 
function, and relationship. Using the analytic framework in an analysis of The 
Farm Adventure for Kids, a popular educational app for pre-school children, we 
found that still images are dominant proportionally and are central in the 
interactive process. We also found that tapping still images of animals on screen is 
the main action, with other screen actions deliberately excluded. Such findings 
suggest that aligning children’s cognitive and physical capabilities to the use of 
mode become the primary consideration in educational app design and that 
consistent attendance to this alignment in mobilizing modes significantly affect an 
educational app’s interactivity, and consequently its reception by young children. 
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INTRODUCTION 

Interactivity influences educational apps’ content, function, relationship, and emotion. 
As Churchill (2011, 2012) elucidates, interactivity affects an educational app’s usability 
and user reception. For instance, how do learners interact with the content and education 
and how do learners interact with one another? For young children, this is especially 
pertinent as learning mostly occurs through immersive interactions such as play and 
experiencing rather than through formal instructions (Hirsch et ah, 2015). It is no doubt 
that unraveling various dimensions of interactivity in educational apps is critical for 
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determining their relevance to young children’s learning. Since most of educational apps 
in Apple Store, Google Play, Windows Store and other mobile spaces are multimodal 
involving modes other than the written text such as sounds, images, and animations, it 
justifies a methodological initiative to understand meaningful inclusion of multimodality 
in engendering interactivity in an educational app. A survey of the current literature as 
summarized in the section that follows further indicates that research has largely 
concentrated on algorithm construct and user feedback; attention to interactivity is either 
anecdotal or non-existent in respect of mode coordination, especially from a social 
semiotics perspective. 

This article is an attempt to address this lack. In what follows, we will first overview the 
main trends and frameworks of educational app design and evaluation to identify 
promises and challenges. We will discuss why interactivity and multimodality should be 
a central concern for educational app development and evaluation from a social semiotic 
perspective. We will then propose an analytic framework to unpack multimodality in 
educational apps and follow it up with a single app analysis to demonstrate the 
framework’s pertinence. In the last section, we will discuss the features that have 
emerged from the analysis in line with the analytic framework, the selected educational 
app, and the key theoretical perspectives to unveil how interactivity is enacted. Because 
educational apps have been widely adopted in formal and informal learning spaces, we 
consider the development of the kind of analytic frameworks as timely and important in 
optimizing apps educational functions. 

Review of the Literature 

Apps are simply miniature computer programs running on mobile phones, tablets, web 
browsers and other smart devices (Noble & Russell, 2013; Roblyer & Doering, 2013). 
They are often commanded by various sensory actions on the screen such as clicking or 
swiping the relevant icon and speaking to an AI (artificial intelligence) assistant such as 
Apple Siri, Google Assistant, or Windows Cortana. The arrival of cloud space has 
further spurred the use and design of apps in the last few years, consequently 
necessitating the evaluation of apps in order to ensure quality. In education where the 
digital divide is narrowing and where such technologies are used in and outside 
classrooms by both teachers and students, app evaluation is particularly important 
(Roblyer & Doering, 2013). As iPads and other tablets have been introduced to the 
classroom, the role and selection of apps and app tools have become an important 
consideration in learning, collaborating and teaching. Churchill (2012) observes 
that they were effective for as: a) support for learning in various settings due to 
portability and fit-for-task suitability, b) support for student engagement and 
quick access to apps that students require for a particular learning task and c) students of 
all levels can use apps, especially for reinforcement and rote learning of basic 
concepts (p. 165). 

Such examples are numerous in education. Comic Life, for example, is used for 
designing comics with free templates. Evernote as an organizational cloud allows users 
to take and share notes of all kinds and automatically syncs them. Free Books has a large 
collection of classic English books. Annotate PDF allows teachers and students to 


International Journal of Instruction, October 2017 • Vol.10, No.4 



Blitz-Raith & Liu 


239 


annotate and share PDF files. Both iBooks and iMovie can be used for digital 
storytelling. The app version of Microsoft Onenote is handy for taking notes and 
creating word documents (Miller and Doering, 2014). Similarly, many apps such as 
Dragonbox are used for addressing subject content such as mathematics and science in 
real time with simulated activities which would not be possible prior to the invention of 
these mobile technologies. As an extension of the digital landscape, mobile learning 
creates alternatives beyond the classroom to make education more accessible and 
inclusive (Banitt et. al, 2008; Hawkins, et al., 2012; Johnson and Marsh, 2014; Kazmer. 
el al., 2012). Emerging learning environments such as flipped classrooms, and 
communal, collaborative, customized, and connected learning spaces have intensified 
the need for “application” in education. By creating a diverse environment in which a 
wide variety of apps are used to increase learning opportunities, mobile learning entails 
many benefits and challenges for learners with varied abilities and preferences (Lai et 
al., 2013; Maich and Hall, 2016; Noble and Russell, 2013; Pegrum, et al., 2013 Pilgrim 
et al., 2012; Wellings and Levine, 2009). Lor instance, many game-based apps have 
been developed to improve the conventional “drill and skill” function in a playful 
fashion. 

Salas (2013) discusses the economic effects and learning revenue growth of educational 
apps. He has recommended sites that have comments and reviews about many different 
types of available educational apps. He has also noted that some of these have been 
designed and written by teachers and are cost free. The goal of the site recommendations 
is to aid teachers in their selection of apps that are classroom friendly and can be 
incorporated actively into curriculum renewal. This highlights the importance of seeking 
solutions to improve the means and methods in selecting useful educational apps. 

The Apple iPad’s inclusion in the classroom has created new opportunities for teaching 
and learning. Murray and Olcese (2011) question whether or not the iPad and its 
attended apps constitute a set of resources that enable teachers and students to do things 
that could not otherwise be achievable in an analogue environment. Some of their 
findings indicate that “the categories offered by Apple are of little use to educators to 
help them make decisions” (p. 46) and that the applications written to run OS devices 
“are woefully out of sync” (p. 48) with modern theories of learning skills that students 
will need in the 21 5,1 century. They suggest that more research become undertaken to 
understand the core principles of how, where, and why people learn in digitizing 
environments. 

Design-based theory reflects the characteristics that make mobile learning environments 
unique compared with traditional learning environments. The interfaces with existing 
technology use characteristics that imply design considerations for specific use and 
purpose. Miller and Doering (2014) consider the following aspects essential in 
effectively developing meaningful mobile learning experiences; 

i. the use of touch screens and gestures, 

ii. contents of use (the affordance of learning across space and time), 

iii. scope (modularity that interfaces with existing technology). 
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iv. data management (the designer needs to identify the type of data the learner will 

want or need to record and how it will be accessed for use at another time), 

v. mode of access (the use of a generic design using HTML.5, CSS3 and JavaScript 

will help to ensure that learning environment is accessible to all learners), 

vi. design scale (the reduction of content clutter and maximization of the 

application’s functionality), and 

vii. incentives (rewards for task completion and encouragement to continue using an 

application) 

Churchill (2012) discusses the ongoing study of apps for delivery via the iPad 
technology and a conceptual model to support learning. His previous studies (Churchill, 
2007, 2008) help to develop two sets of guidelines for the design of conceptual models; 
namely, presentation design and design for learning uses. Presentation design links the 
features of design such as multiple screen, visual, color, audio, animation, multimodal 
features of the app whereas learning uses design implies the use of a conceptual model 
supports pedagogical learning in a real environment. 

Hirsch et al. (2015) in their study on the use of apps for younger children discuss four 
evidence-based pillars of learning essential for educational app design: active, engaged, 
meaningful, and socially interactive. They were considered instrumental to creating 
"attention" or "motivation"; however, motivation is a meaningless term without tapping 
into these elements. An important area of learning is known as “active minds on” 
learning “where the learning activities are designed to engage the student and to build 
and reshape their thinking and understanding as a natural consequence of their 
experiences and interactions within learning environments” (Hirsch et al., 2015, p.8). 
The types of “minds on” learning include activities such as interpretation and the ability 
to translate words to mental images. App designers may use symbols to support active 
cognition. This may be useful for mathematical and musical apps. It seems that apps’ 
inbuilt interactivity enables the user to explore more information about a subject or to 
reach a higher level of learning. The belief in the power of design has compelled Hirsch 
et al. (2016) to conclude that apps that engage these pillars within the context of a 
learning goal are most likely to be truly educational. 

From a social semiotic perspective, however, such a conclusion looks over-optimistic, if 
not questionable. A social semiotician would consider an educational app essentially a 
multimodal text that creates and transacts meaning in context (Kress, 2010, 2015, Kress 
and Van Leeuwin, 2015). As a key player in creating multimodal text, modes are 
socially and culturally shaped in enabling resource to make meaning. Modes can include 
but are not limited to, image, writing, layout, speech, and animation. Their typification is 
often situated in relation to research design and participants. Writing, for instance, is 
realized by syntax, grammar, lexis, punctuation, font, color and size. Speech is enabled 
by loudness, pitch and pitch variation, tonal and vocal quality, length and silence. Image 
utilizes the positioning of elements in a framed space (e.g., size, color, shape, icons, 
lines, circles); and in the case of animation, the movement of a succession of images. 
These differences in modes mean that they can be used to do different semiotic work 
with different resources in different ways. Modes also entail different affordances, 
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potentials and constraints in the making of meaning. Burlamqui and Dong (2015, p.308) 
regard affordance as “cues of the potential uses of an artefact by an agent, in a given 
environment”. In app development, various modes are remixed to make meaning 
through the affordances of the modal resources. Therefore, meaning is orchestrated 
through the selection and configuration of modes. In other words, it is the interactivity 
between the modes and their characteristics that is significant for meaning making. 

Interactivity describes the affordance of a medium or the media’s potential ability to 
allow the user to exert an influence on the content (Jensen, 1998). Furthermore, the 
analysis of interactivity can offer insights into a user’s perceived interaction (McMillan, 
2009) and the ways in which the users and images respond to meaning 
making. Educational apps naturally depend on interactivity in order to enable user 
participation. Links and buttons are not only signs of making meaning, but also sites of 
action producing. Social semiotics can be used to discuss the relationship between their 
form, the actions required to activate them, and the effects that can be produced. In this 
regard, it is imperative to develop a framework that can help unfold interactivity in 
educational apps. 

METHOD 

This study adopts multimodal discourse analysis (MDA) as its research method. MDA, 
as an emerging member of discourse analysis (DA) family, focuses on understanding 
how non-discursive modes such as image, color, sound, and movement are employed as 
semiotic resource in creating meanings (Baldry & Thibault, 2006; Kress, 2010). In what 
follows, we will describe three aspects of this MDA study; namely, 1) the criteria for 
selecting a suitable educational app as the data, 2) the process that prepares the selected 
educational app for the study, and 3) the analytic framework for uncovering interactivity 
among the modes involved in the educational app. 

Data collection 

The data for this study was collected from a popular children’s educational app. Farm 
Adventures for Kids. According to Baldry and Thibault (2006), text selection in MDA 
should take into account the nature of text and the context around the kind of text. As 
such, to inquire into the applicability of the above-discussed MDA framework, 
we developed four criteria in selecting the educational apps; namely, a) the selected 
educational app should work on multiple mobile operating systems, b) it should relate to 
a curriculum, c) it should be affordable, and d) it should be popular. The app we have 
finally chosen for the analysis. Farm Adventure for Kids , fulfils these selection 
criteria. The app is aimed at young children who may not have the opportunity to visit 
farms or see farm animals. It is available as a free app in both Apple Store and Android 
Play Store. It has a four star (out of five) rating, with more than one million downloads 
and over 700 in-store customer reviews. Since the contents of the app include learning 
skills universally stipulated in early years curriculum such as learning the alphabet, 
spelling and recognition skills, and simple math such as counting, it may be used as a 
complementary learning tool or space in the school classroom. As Figure 1 below 
illustrates, the app visualizes what a farm looks like and what kinds of animals and foods 
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are raised on an animated journey. Its What’s New section in Apple Store and Android 
Play Store indicates that its developer believes that it would help create mentally and 
physically adventures that lead to active learning and personalized discovery. 



Figure 1 

A Screenshot of Farm Adventure for Kids 


Data transcription 

Data collected from the selected educational app was processed and labelled in line with 
the app’s progress from level one to level ten. Each level of the apps was further 
annotated by Content, Learning Activity, and Stage (see Table 1). This annotation 
would prepare the data systematically to accommodate the nuanced analysis of modes. 
Table 1 summarizes the 10 levels of activities in the app. The table indicates that levels 
one and two are designed as learning activities and that levels three, four and five 
involve games that reinforce what has been learnt. The first two levels involve basic 
learning skills for the young learners. They cover the contents that are taught at school 
and introduce more complex concepts as the young learner’s skillset improves. Level six 
uses named and brightly colored shapes that can also be dragged and dropped to 
complete puzzles. Level seven is used for dropping objects, whilst level eight increases 
the scale of difficulty. Level nine is composed of games and level ten tests what has 
been learnt from completing the earlier levels. The app uses games for counting, 
spelling, recognition of color, shapes, and sounds to enrich young learners’ 
understanding. 
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Table 1 

Contents of Farm Adventures for Kids 


Level 

Content 

Learning Activity 

Stage 

1 

Alphabet 
animals, 
numbers, fruits, 
vegetables 

The characters appear on the screen moving from right 
to left after tapping 

The name of the animal and the sound it makes is heard 
as are the other animated artefacts 

Introducing 

skills 

2 

Colors 

The names of different colors are learnt 


3 

Sort Games 

Vegetables are sorted and placed in their basket. 

The names of the different vegetables are learnt 


4 

Labyrinth 

Eggs that will be used to form words are collected 

Consolidating 


Games 

The spelling of the names of the animals is learnt 

skills 

5 

Jump Games 

Fruits, vegetables, letters of the alphabet and numbers 
are collected 

This is a learning and dexterity testing game. 


6 

Match Games 

Shapes in the boxes are arranged to form a new figure. 
The names of shapes are learnt 

Complicating 

7 

Drop 

The names of fruits and vegetables are learnt 

This reinforces what has been learnt 

skills 

8 

Shaped Fence 

A game for learning numbers and shapes 


9 

Play games 

Connecting colors to the animals 

These are dexterity games; 

Advancing 

10 

Quizzes 

Knowledge and skills that have been learnt are tested. 
This is used to reinforce what has been learnt 

skills 


Data analysis 


To analyze the selected educational app, we have developed a MDA framework as 
described in the following. We will explore its application and possible extension 
through the analysis of this educational app. We have come to a realization that 
developing an educational app-specific analytic framework be more useful than a 
blanket adoption of an existing model. We envision the analytic framework with the 
following three attributes. First, it should be able to describe the modes involved and 
their occurrence. Second, it should be able to describe the functions and roles of the 
modes involved. Third, it should be able to describe the relationship between the modes 
involved. Inclusion of these attributes is anticipated to provide a semiotic delineation of 
the connections between multimodality. It is also anticipated that the analysis may help 
generate tangible categories and scales that can be exploited in further quantitative 
investigations. We focus our attention mainly on five primary types of mode, namely, 1) 
still image, 2) animated image, 3) movement, 4) sound, and 5) written text. To ensure 
the analysis’s coherence and feasibility, other modes and modalities such as color, 
typography, and shape are excluded from the present study, considering mode 
complexity, as well as the unsettled theoretical debate around modes, modality, and 
medium (e.g., Bezemer & Jewitt, 2010). 
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Figure 2 

A MDA framework for educational apps analysis 

As Figure 2 illustrates, the proposed multimodal analytic framework comprises three 
layers: mode multiplicity, mode function, and mode relationship. The first layer of the 
framework, mode multiplicity, examines occurrence of the involved in creating an 
educational apps. This decision is inspired by Norris’s (2004) view that when all the 
modes that participate in the modal interaction are interconnected, multimodal 
interaction can be described by examining attributes such as modal density, intensity, 
complexity, and weight. Norris’s categorizations are certainly useful in generating 
interpretations; yet, they are of little use to hands-on analysis of the mode presence 
in educational apps, as conceptual terms rather than as tangible variables or factors. 
Mistaking conceptual terms for variables in this respect may result in fatal data 
processing errors, especially when an analysis is applied to developing large-scale 
databases or is adapted as computational models. In lieu of Norris’ model, mode 
multiplicity in this framework focuses on a) the number of modes; i.e., how many modes 
are used, and b) the frequency of modes; i.e., how often does a mode recur. Both are 
attributes that can be numerated to represent Norris’ conceptual terms such as modal 
density, complexity, intensity, and weight. It should be pointed out there that other 
factors such as distance and cluster are equally useful but are temporarily excluded in 
consideration of this article’s purpose. 

The second layer of the framework, mode function, identifies what a mode can do in 
enabling an educational app’s interactivity. In his analysis of websites, Adami (2013) 
adapted the three metafunctions of Systemic Functional Grammar (Halliday, 1994; 
Kress, 1996; Martin, 1993), namely, ideational, interpersonal, and textual, to account for 
the interactive meaning potentials of website as digital text. Adami’s adaptation pertains 
to the present study on educational apps in that both websites and apps are semiotic texts 
that instantiate discoursal attributes inclusive of interactivity from a social semiotics 
point of view. Interactions between iPad users and iPad apps are comparable to those 
between web surfers and websites. In this sense, the attributes developed by Adami 
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(2013) can be refined to accommodate mode functions required in the present analytic 
framework. They include: (a) a meaningful form; i.e., what does a mode look like, (b) an 
action; i.e., what does a mode do, and (c) an effect; i.e., what is a mode’s impact. In 
other words, the function is realized by the form that a mode takes, by the action that is 
needed to produce an effect, and by the effect is that is produced from this combination 
of mode functions. 

The third layer of the framework, mode relationship, examines how modes are related to 
one another in an educational app. From an orthodox semiotic perspective on 
multimodality (Burlamqui and Dong, 2014; Kress, 2010, 2015), the concept of 
anchorage must be evoked in locating the modes that anchor the meaning of all 
participating modes. Adhering to this perspective seems reasonable but in practice is not 
productive or illuminating in that 1) the relationships among the modes around the 
anchor can hardly be usefully delineated against a reliable framework, and 2) it may be 
interpreted as patriarchal when a particular mode is predetermined to dominate the 
entire meaning making process. Text analysts on the other hand offer a different 
approach by focusing on how meaning is (in) coherently and (in) effectively engendered 
in a text. Seeing language as a system, several textual linguists (De Beaugrande, 1981; 
Lemke, 2005; Mann & Thompson, 1988) have managed to identify rather elaborate 
textual relations, devices, and principles. While acknowledging their contributions, 
rather than becoming entangled in the complexity of textual systems, we have decided to 
focus on identification of modes that make the meaning explicit and of modes that create 
details or support. We label the former the principal mode and the latter the subordinate 
mode. We are conscious that other kinds of mode relations such as conditional and 
concessional are equally important and will return to them in further studies. It should be 
noted that methodological validity and reliability is not a point of discussion in this 
exploratory, qualitative study but should be addressed carefully in future corpus- 
oriented larger scale analyses that involve layers of variables. 

FINDINGS 

The app’s interactivity is explored in the following three levels of multimodal analysis. 
Conscious of the space, we only present the analysis of the app’s first four levels in the 
tables that follow. 

Mode multiplicity 

Table 2 summarizes the first four levels of this analysis. The quantity and frequency of 
modes are described and numerated. An X is placed to indicate a mode’s absence. 
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Table 2 


Mode multiplicity 


Level Mode 

Quantity 

Frequency (Times) 

1 Still image 

2 

Tapping on the screen and tapping on an animal 

Animated image 

7 

Once per animal 

Movement 

7 

Once per animal moving from right to left 

Sound 

7 

Animal noise and narrator clearly says animal name, 
background music 

Written text 

7 

Instructions and name of each animal 

2 Still image 

1 

Tapping on the screen and tapping on a color 

Animated image 

10 

Once per animal 

Movement 

7 

Once per color being removed 

Sound 

7 

Narrator clearly says colorr name, background music 

Written text 

X 

X 

3 Still image 

2 

Tapping on the screen and tapping on a vegetable 

Animated image 

3 

Once per vegetable 

Movement 

7 

Once per animal moving from right to left 

Sound 

7 

Narrator clearly says vegetable name and background music 

Written text 

X 

X 

4 Still image 

2 

Tapping on the screen and tapping once for each letter of the 
alphabet 

Animated image 

7 

Once for each letter of the alphabet 

Movement 

7 

Once per animal and once for each letter of the alphabet used 
to spell the animal name 

Sound 

7 

Narrator clearly says each letter and the name of the animal 
and background music 

Written text 

7 

Name of the animal 


The analysis shows that each level has at least four modes; namely, the still image, 
animated image, movement and sound. The primary mode across the four levels are 
animated image of the screen and animals. Movement is manifested in signs such as the 
dog wagging its tail and colored blocks falling. Sound appears across the four levels in 
two forms: a) background music and b) animal sounds. The same background music has 
been used throughout all levels of the app. Farm animal sounds such as the dog barking 
or the horse neighing and the narrator are accompanied by enunciation of animal names 
in words and letters. The written text which appears in levels 1 and 4 adds to the modal 
complexity and density of the interlinked nodes. 

Mode functions 

This part of analysis further examines the functions of the identified modes across the 
levels of the app. Table 3 summarizes the mode functions presented in the first five 
levels of the app. 
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Table 3 

Mode Functions 


Level 

Mode 

Form 

Action 

Effect 

1 

Still 

Physical action 

Animated animals 

Each animal’s name appears and is 


image 

tapping by index 
finger 

move from right to 
left 

spoken by narrator. Animal makes a 
noised e.g. dog barks 


Animated 

Animated 

Animated animals 

Each animal’s name appears and is 


image 

animals 

move left to right 

spoken by narrator. Animal makes a 
noised e.g. dog barks 


Still 

image 

Windmill 

Appears in frame 

Windmill blades slowly move 

2 

Still 

Physical action 

Colored blocks 

Each color is spoken y narrator as they 


image 

tapping by index 
finger 

appear 

appear 


Still 

image 

Colored shapes 

Colored blocks 
appear 

Each color is spoken y narrator as they 
appear 

3 

Still 

Physical action 

Assorted vegetables 

Each vegetable is named by narrator 


image 

tapping by index 
finger 

moving right to left 
above basket 

and is dropped into a basket if tapped 
in time 


Still 

Animated 

Assorted vegetables 

Each vegetable is named by narrator 


image 

vegetables 

moving right to left 
above basket 

and is dropped into a basket if tapped 
in time 

4 

Still 

Physical action 

Gathering assorted 

Each letter is named and spoken by the 


image 

tapping by index 
finger 

letters of the 
alphabet 

narrator until the animal’s name is 
written 


Still 

image 

Animated animal Gathering assorted 
letters of the 
alphabet 

Each letter is named and spoken by the 
narrator until the animal’s name is 
spelt 


The tables show that the mode in each level is the still image which takes form of a 
physical action realized by tapping the index finger on the touchscreen. This produces 
the action of movement of the animated image with the effect that the animal’s or 
object’s name appears and is articulated by the narrator when the image is tapped. In 
level 3 there is additional movement of the fruit/vegetables dropping into a basket after 
tapping. Level 4 is more complex in that immediately would an animal’s name appear in 
the written text when each letter of the alphabet is spoken 

Mode relationships 

This part of analysis examines the relationship among the present modes in the app by 
identifying which modes are principal and which modes are subordinate, as Table 4 
below summarizes. 
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Table 4 

Mode Relationships 


Level Principal mode 

Subordinate mode 

Elaboration 

1 

Animated image of animals 

Movement 

Dog running, cat jumping 



Sound 

Dog barking, horse neighing 



Written text 

The name of the animal appears and is 
clearly spoken 

2 

Animated image of color 

Movement 

Colored shapes move and stack 



Sound 

The name of the color is clearly spoken 

3 

Animated images of artefacts 

Movement 

Vegetables moving from right to left 

4 

Animated image of alphabet 

Movement 

Each letter jumps and moves to spell 
the name of an animal 



Sound 

The names of the letters and the animal 
appears and is clearly spoken 



Written text 

The name of the animal appears and is 
clearly spoken 


The principal mode in each level is the animated image and the common subordinate 
mode is movement. The movement is seen in the animated artefact such as the dog 
wagging its tail, the vegetables jumping into the basket. Sound is a subordinate mode of 
levels 1, 2 and 4, adding to the complexity of the mode relation typified by the 
intonation and enunciation by the narrator while the overlay of simple and repetitive 
background music occurs in all levels. The written text is a subordinate mode of levels 1 
and 4. As the levels increase in complexity, more common subordinate modes are 
visible in every level such as movements and spatial effects, probably to enhance the 
principal mode. 

DISCUSSION 

We discuss in this section several features that have emerged from the analysis in 
relation to interactivity. First, it appears that that still images play a significant role in 
creating the app’s interactivity. More than 100 still images appear across the five levels 
and each of the images occurs four times. For example, the farmyard, the farmyard map 
(layout), the picture of a pointing index finger as instruction to tap the screen and 
additional tasks at the end of each level. Still images, though not as animated, function 
not only as background of the app to delineate the farm activities but also to facilitate 
collaboration among other modes. To some extent, this resonates with Bezemer and 
Kress’ (2008) observation that a visual and verbal depiction of the same thing can be 
thought of as two different semiotic representations of the same phenomenon to 
engender a simultaneous semiotic relationship of identity and differentiation. Lemke 
(2002. p. 3) explains “that it is this essential incommensurability that enables new 
meanings to be made from combinations of modalities in text”. As the meaning-making 
affordances of image and text differ, they interact synergistically in the construction of 
meaning for young children. From an evolutionary cognitive perspective (Gazzaniga, 
2004; Heyes, 2012), this semiotic saturation of still images also underscores the 
importance of visual perception in human learning, particularly in creating attention, 
sustaining motivation, and scaffolding tasks. Knowingly an essential avenue to 
empowering young children in learning, it has been long neglected due to the dominance 
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of written texts and the print as well as the cost in mass production of images. It seems 
inevitable that effective educational apps development for children have to embrace a 
renaissance of “the visual’’ in planning and design. 

Further, we noticed that among all the possible actions afforded by smart devices, 
tapping is the most frequently intended action. Even more interestingly, tapping is 
usually initiated by a still image of a farm animal and then followed by a typical sound 
effect and/or movement of the animal. An example of this is the tapping the start screen 
in level one and animated animals start moving from right to left across the screen. 
Tapping on the animal produces the spoken name of the animal as well as the sound the 
animal makes. For example, the name of horse appears as texts and is spoken and then 
the horse neighs. On the surface, this responds to the design-based approach (Miller and 
Doering, 2014) which stresses the place of the touchscreen’s materiality and affordance 
in facilitating app-based activities. However, it does not explicate systematically or 
coherently how tapping as an action bridge the transition from a mode’s form to its 
effects. It is likely that a touchscreen action grammar is in need of development to look 
into this kind of transition, semiotically. Since educational apps on mobile smart devices 
are mostly enabled by the touches screen, a delineation of users’ common hand 
movements on the screen is necessary in developing this semiotic grammar; they may 
include 1) tapping (fast or slow), 2) swiping (left, right, up, or down), and 3) circling 
(long press or light touch). It is also necessary to examine whether these actions involve 
just left or right hand or both hands coordination. Whether these screen actions are 
present and how often they occur are important indicators of an educational app’s 
interactivity. 

As Kress and Van Leeuwin (1996, 2006) state, image, color, music, typography and 
other visual modes are similar to language; in other words, they can simultaneously fulfil 
and realize broad communicative functions in a similar manner to language in that image 
and other visual modes have the capacity to form texts. Kress (2003) believes that 
different logics govern the mode of written language and that of visual image. Written 
text is governed by the logic of time or temporal sequence whereas visual image is 
governed by the logic of spatiality, organized arrangements, and simultanicity. In other 
words, meaning is derived from position in the temporal sequence of written text, 
whereas meaning is made from the spatial relations or grammar of visual images (Kress 
& Van Leeuwin, 1996). Although this claim has been challenged (Prior, 2005), it does 
offer a useful angle in understanding why image tapping is a designated feature of the 
chosen app. 

Further from there, we noticed that animal images are also central in constructing mode 
relationship. Still images of the farm animals are the most frequent among the identified 
principal modes while other modes such as written text and hyperlinks are either non¬ 
existent or subordinate. An affordance point of view (Gibson, 1979) may suggest that 
animal images engender not only visual imagination but also desires to experience and 
restmcture visual representations. The concept of anchorage crystallizes the kind of 
image-to-image or image-to-text relationship in which the embedded image functions as 
a semiotic anchor. Other anchors such as links and buttons enable users to act upon the 
site, thus obtaining some effects. This aligns closely with the trend that the written text is 
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no longer the central mode of representation in learning materials such as textbooks, 
web-based resources and teacher-produced material (Bezemer and Kress, 2010). 
Moreover, moving images are found to be increasingly prominent as carriers of 
meaning. That is another indicator that educational app design should embrace more 
vigorous multimodal text analysis as such to understand the role of interactivity. 

This testifies to a reality that although interactivity is a widely used transdisciplinary 
term, it is still perceived as under-defined and undertheorized as if it is an elusive 
concept employed to refer to different phenomena (Adami, 2013; Bucy, 2004; 
McMillan, 2002; Sundar, 2004). Kress (2003) argues that interactivity is a remix of 
interpersonality and hypertextuality. Interpersonality has an effect on social power 
directly whereas hypertextuality has an effect on semiotic power and through that on 
social power less immediately (Kress, 2003). This remix encompasses mode 
multiplicity, function, and relationship as described in this article; e.g., images can be 
combined with other modes and actions while spatial and temporal modal activity adds 
to the complexity of their inter-semiotic relationship. Even though the kind of mode 
relationship uncovered in the analysis is singular, it opens up doors to more 
comprehensive, nuanced examination of inter-semiotic relations afforded by multimodal 
educational apps. 

Most importantly, we noticed from the analysis that consistency seems to serve as a 
coordinating principle in achieving textual coherence when the app is considered as a 
text from a social semiotic point of view. For example, throughout the entire app, still 
images are consistently employed as the dominant mode and tapping as the main action, 
which may be indicative of the designer’s adherence to children’s cognitive ability and 
physical coordination. A simple mode relationship that anchors still images as the 
principal modes and other modes as subordinate aligns with children’s interpersonal 
development, albeit being overly simplistic. Children’s sense of self, their perception of 
others, and their connections to the beings around them operate in a relatively less 
complex fashion and should be explicitly addressed in designing this kind of educational 
apps. Consistency in this regard is paramount in that it ensures a thorough attendance to 
the overarching context in educational design. 

CONCLUSION 

We may well conclude our analytical discussion by highlighting the importance of mode 
selection, arrangement, and connection in creating an educational app’s interactivity and 
its user reception. We suggest that the key to an app’s success may not lie in the 
profusion of modes but in what modes are included and how they are mobilized and 
interconnected. Such configurations of modes may not simply be determined by the 
medium (the device), the ecology (the app system), the modes available, or the 
technology. Instead, they may be shaped by insider’s understanding of young users— 
their cognitive ability, their physical ability, and their habitus such as interests and 
preferences. Apparently, even though the analytic framework employed in the analysis 
is useful, further adjustment is needed in order to undertake multiple-case or large-scale 
analysis. For instances, mode functions should spell out various groups of forms, 
actions, and effects. Various types of subordinate relationship can be further identified 
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and examined. It is inevitable that this analytic framework be refined beyond the sole 
confinement of design on its sociological inquiry voyage of educational apps. 
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Turkish Abstract 

KiRiik Gocuklar Rin interaktif Egitim Uygulamalan: Bir Goklu Mod Analizi 

Bu makale sosyal gostergebilim yaklasjimlanndan yola qikilarak, etkile§im yaratmada 119 katman 
modelini incelemek Rin ?okluk, i§lev ve ili§ki olmak iizere 90 k modlu bir analitik 9 er?eve 
onermektedir; Analitik 9 er 9 eve, okul oncesi 90 cuklar Rin popiiler bir egitim uygulamasi olan The 
Farm Adventure for Kids'in bir analizinde kullamlarak, hareketsiz goriintulerin oransal olarak 
egemen oldugu ve interaktif siire 9 te merkez oldugunu tespit edilmRtir. Ayrica, hayvanlarm 
hareketsiz goruntiilerini ekranda izlemenin ba§hca eylem oldugu ve diger ekran Rlemlerinin 
kasitli olarak hari 9 tutuldugunu ke§fedilmi§tir. 

Anahtar Kelimeler: uygulama, mod, 9 oklu mod, etkile§im, sosyal gostergebilim, metin 


French Abstract 

Interactivity dans Apps Educatifs pour Jeunes enfants: une Analyse Multimodale 

En dessinant d'approches de semiotique sociales, cet article propose un cadre analytique 
multimodal pour examiner trois couches de mode dans l'engendrement de l'interaction; a savoir, 
multiplicity, fonction et relation. En utilisant le cadre analytique dans une analyse de l'Aventure 
Agricole pour des Enfants, un app educatif populaire pour des enfants d'ecole matemelle, nous 
avons constate que les images fixe sont dominantes proportionnellement et sont centrales dans le 
processus interactif. Nous avons aussi constate que le decouvrant des images fixe d'animaux sur 
l'ecran est Taction principale, avec d'autres actions d'ecran deliberement exclues. 

Mots Cles: appification, mode, multimodalite, interaction, semiotique sociale, texte 
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German Abstract 

Interaktivitat in Bildungs-Apps fiir Kleinkinder: Eine multimodale Analyse 

Ausgehend von sozialen Semiotik-Ansatzen, schlagt dieser Artikel ein multimodales analytisches 
Framework vor, um drei Schichten des Modus bei der Erzeugung von Interaktion zu untersuchen; 
namlich Multiplizitat, Funktion und Beziehung. Mit dem analytischen Rahmen in einer Analyse 
von The Farm Adventure for Kids, einer beliebten Bildungs-App fur Vorschulkinder, fanden wir, 
dass noch Bilder proportional dominieren und im interaktiven Prozess zentral sind. Wir haben 
auch festgestellt, dass das Tippen von Bildern von Tieren auf dem Bildschirm ist die wichtigste 
Aktion, mit anderen Screen-Aktionen bewusst ausgeschlossen. 

Schliisselworter: angabe, modus, multimodalitat, interaktion, soziale semiotik, text 

Malaysian Abstract 

Interaktiviti dalam Aplikasi Pendidikan untuk Kanak-kanak Kecil: Analisis Multimodal 

Menggambar dari pendekatan semiotika sosial, artikel ini mencadangkan rangka kerja analisis 
multimodal untuk mengkaji tiga lapisan mod dalam interaksi yang mencetuskan; iaitu, 
kepelbagaian, fungsi, dan hubungan. Menggunakan rangka kerja analisis dalam analisis The Farm 
Adventure for Kids, aplikasi pendidikan yang popular untuk kanak-kanak prasekolah, kami 
mendapati imej pegun dominan secara proporsional dan menjadi pusat dalam proses interaktif. 
Kami juga mendapati bahawa mengetuk gambar binatang di skrin adalah tindakan utama, dengan 
tindakan skrin lain sengaja dikecualikan. 

Kata Kunci: aplikasinya, mod, multimodaliti, interaksi, semiotik sosial, teks 

Russian Abstract 

HHTepaKTHBiiocTb O6pa30BaTejibHbix llpii.iO/Keniiii , 1 , 1 a /ieieii Mjiaqmero Bo3pacTa: 
Mya binMO.ia n. iu.i ii AHajiH3 

OcHOBWBaacb Ha qoqxoqax coqiiaabqoii ccmhothkh, b 3toh CTaTbe qpeqaaraeTca 
MyjibTHMoqajibHaa aqaaiiTiiaecKaa CTpyKTypa qua H3yaeqqa Tpex ypoBHeii Moqeaeii b qpoqecce 
(jtopMHpoBaqua B3aiiMoqeHCTBHa; a HMeqqo, MqoacecTBeqqocTb, (jtyqKqqq h B3aHMOCBH3B, 
HcqoJib30Baqqe aqajiHTHuecKOH ochobbi qpn aqajiH3e (jtepMepcicoro qpHKJitoueqqa qaa qeTeii, 
qoqyaapqoe yue6qoe qpnaoaceqqe qua qeTeii qoniKoabqoro B03pacTa, mm o6Hapyacqaq, hto 
qeqoqBtDKHbie H3o6paaceqqa qoMuqqpyioT qpoqopqiioqaabqo h 3aqqMatoT qeqTpaabqyto poab b 
qqTepaKTHBqoM qpoqecce. Mbi Tatcace o6Hapyacqaii, hto qaacaTHe qeqoqBiiacqbix H3o6paaceqqq 
aatBOTqbix qa stcpaqe aBJiaeTca ocqoBqMM qeiicTBHeM, qpn stom qpyrqe qeiicTBHa sxpaqa 
co3HaTeabqo HCKqtoneqM. 

KatoueBMe CaoBa: qpqaoaceqqe, peatHM, MyabTHMoqaabHOCTb, B3aHMoqeHCTBiie, coquaabqaa 
CeMHOTHKa, TeKCT 
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